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Why is it so hard to talk about 

sound? 
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Because sound drives EVERYTHING 


Game (and reality) elements 

• Visuals - Huge vocabulary of nouns 
and adjectives 

• Characters - Similarly huge 
vocabulary of social descriptors (age, 
physique, gender, race, social status, 
behavior). 

• Physics - Rules about how elements 
interact - very linear vocabulary (even 
with non-linear rules) over specific 
time frames. 

• Speech - strict linguistic rules. 


• Sound is about events in time across 
large time frame (milliseconds to 
minutes or more). 

• Non-scientific vocabulary highly 
subjective. 

• Deeply tied to emotional and 
unconscious states and reactions. 

• Perception of sound works at pre- and 
subconscious time frames. 

• Enormous cultural and demographic 
differences color perception of 
situations and events. 
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What is a game? 

• A reality in a box 

• Reality is built from our reactions to input 

• Inputs come from our six senses 

• But for the most part, game realities have to 
rely on only two senses: vision and hearing. 

• In a well designed game environment, the brain 
fills in the rest. 
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Reality vs. the brain 
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Maps of the world 



Diffusion tensor images of connectivity of Cortical mapping of somatosensory 

memory, vision, language, arousal (Liu et al) (left) and motor output (right) in 

human cortex (Penfield). 



o° 

Azimuth 


Owls (and humans) make 
spatial maps with sound 
that guide their vision 
(Knudsen) 


• Psychophysics - mapping the physics of 
the outside world onto the psychological 
internal representation of the world. 
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Lining up the maps 



Subcortical regions bring 
early sensory data into 
alignment into common 
coordinates (Pittl et al) 



Decision making areas like the pFC 
integrate senses on a conscious level 



But the map is not the 
territory (sensory 
homunculus) 


• The brain makes maps by bringing different sensory/motor phenomenon into a 
common register, using mechanisms like attention. 






GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


Navigating some maps takes more time than others 
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• There is no place in the human brain you • 

can't get a visual response. 

• Visual recognition takes a minimum of • 

0.25 seconds, usually more like 0.75 

seconds or about the speed of conscious • 

thought 


More discrete projections from the ear throughout the 
brain (although goes to almost as many places). 
Complex feature processing and responding requires 
very little brain at all. 

Recognize sounds in 0.05 seconds 

Differences easy to detect down to 0.0003 seconds 
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Sensory speeds - Never quite "now" 

• Vision (150-400 msec) 

• Hearing (50-200 msec) 

• Touch: 

• Deep pressure (proprioception) skeletal muscle 80-120 m/sec 

• Light pressure (mechanoreceptor) 53-75 m/sec 

• Pain/temperature 5-35 m/sec 

• Smell (500-2500 msec + effusion time) 

• Taste - extremely variable depending on the components, often not 
overlapping but close to smell 

• Balance - (20 ms to eye correction but up to 480 msec until perceptual 
onset.) 
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Why is hearing different from other 
senses? 


• Hearing is a universal sense. 

• Hearing is the fastest sense. 

• Even though humans are "visual" animals, our 
sense of hearing provides us with our primary 
handle on the environment, out of line of sight 
and even in the dark. 
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Frequency 

- Place of maximal vibration along basilar membrane 

- Which hair cells respond 

- A tonotopic map in the cochlea 
Period 

Auditory nerve fibers measure the time interval between individual cycles in 
the sound 

Neurons "phase-lock" cleanly up to 1500 Hz, "Volley" with other nerve 
fibers between 1000-5000 Hz. 
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The Quick or the Dead 



"The ear doesn't blink" 

No blind spots - Works in the dark - Works when asleep. 

Hearing tells you what is happening in the world: 

The environment you're in shapes the sound. 

The shape of your ears and head shapes the 
sound. 

Your age, health and personal history shapes what 
you can detect. 

Within 50 milliseconds: 



Where it is. 

What it is. 

Who it is. 

Should you run away from it. 
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High speed auditory processing underlies our 
perceptions of complex properties of the 
environment. 

• Material 

• Density 

• Weight 

• Power 

• Emotional meaning 

• Condition of item 

• Time/space/place 
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Is Seeing Believing? 







GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


The "speed" of hearing is why there are so 
few auditory illusions 

• Auditory illusions are rare and usually subtle. 

• Require deliberate manipulation usually by 
technical means. 

• Pitch based illusions depend on very complex 
sounds with multiple harmonics. 

• But they can be very revealing about how the 
brain processes complex sounds. 
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Psychophysics & the Non-Linear Mind 

• Psychophysics is how we go from the physics of 
sensation to the psychology of perception. 

• Hearing is more than the physics of sound. 

• Your ears are not digital receivers. 

• Your brain isn't even CLOSE to linear. 

• We sense everything within our range, but we 
pick and choose what we perceive. 
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The signal and the noise 



• To listen, you need to differentiate between signal and noise. 

• Listening implies communication: a sender, a receiver and a 
signal. 

• Signals generated by breaking up noise into temporal or spatial 
patterns. 

• To identify and understand a signal (and tune out the noise), you 
need to pay attention. 
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Attention is about synchronized input over time 


Sending 

neurons 


No synchrony (noise) 
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Receiving 

neuron 



Figure 3. 


Desimone 2007 


Synchrony in a subset of neurons 
Sending Receiving 

neurons Signal neuron 
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FEF 

(Frontal Eye Fields) 
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(Lateral Prefrontal Cortex) 


V4 

(Visual Area V4) 


Bottom-up Control of Attention 

Attention is grabbed by salient stimuli: stimuli that 
have high intrinsic value (such as a fast moving 
object). 

Automatic highlighting of salient stimuli: 

Pulvinar — - — 

& Thalamic Nuclei 

£*£\ ’3 RLvs 

Top-down Control of Attention 

Attention is guided by internal values and goats, allowing one to select 
stimuli that are relevant to the current task. 

Top-down signal originate In frontal cortex and flow towards 
posterior cortex: 

FEF & MT 

► LIP LIP ► M 1 V4 ► V2 ► VI 

LPFC (Visual Area) 

Buschman and Millet. Saalmaim el af Buffalo ef at. PN AS 201 0 

Soence 2007 Science 2007 

This is likely due to competitive interactions within 
a cortical region (see Figure 2) and flows in a 
bottom-up manner: 

LIP ► LPFC fr> FEF 


Current Opinion in Neurobiology 


Signals align in time based on overlap of • 
features 

The greater the synchronization the easier • 
it is to shift attention and the harder it is # 
to ignore the feature. 


Two types of attention: Top down (task 
driven) and bottom up (stimulus driven). 

Each has separate pathways. 

Final target is the prefrontal cortex. 
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Why paying attention is hard. 
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From Tregellas et al, 2012 


• Attention span as measure of work 
brain is doing on tasks. 

• Attention NARROWS your input. 

• Tremendous natural variation on 
depth and span of attention. 

• Extended listening highly energy 
intensive. 

• Extended listening fighting hundreds 
of millions of years of evolution. 
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Visual 


vs auditory attention 



Finding Waldo can take minutes due to slowness of visual search in a noisy field. 

Cocktail party effect (auditory equivalent of “Where’s Waldo?”) shows your ears 
can find relevant sounds in milliseconds. 


# 


Cocktail party 


4 


Your Party? 
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Auditory Attention 


(a) Perception 

Auditory streams 


(b) Biological substrate 

(Where) 


(c) Computations 

(How) 
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Current Opinion in Neurobtology 


• Auditory attention is different from visual attention even though it feeds into 
similar pathways. 

• Auditory input 5-20X faster than vision. 

• Better at stimulus driven, can be harder to sustain for task driven. 
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Listening with More Than Ears 
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Driver & Noesselt, 2008) 

• We think about listening as if it's something locked into our ears. 

• In reality, there are few places in the brain that respond to only one sense. 

• Attention operates across all the senses, and the narrowing of focus can use more than one sense. 
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Multisensory attention 

• When objects are perceived by multiple sensory 
systems, they increase measurable attentional 
loads. 

• Multisensory attention, especially when 
consciously attended to, calls in areas of the 
prefrontal cortex where it plays a part in decision 
making. 

• This slows your brain down a lot and makes it 
amenable to errors and illusions. 
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The McGurk Effect: Vision guides what you hear 
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Attention & Cognition 

• Attention is a range limiting process 
(yielding apparent resolution 
enhancement). 

• Attention reinforces familiarity and 
creates expectations 
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The roles of context and expectation 


Rain on a roof? Bacon cooking in a pan? 
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emotional response 
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Emotional Listening 

• Sound is the most powerful driver of emotions. 

• Music is an emotional language. 

• We rapidly and without conscious volition respond to and identify 
the emotional meaning behind some sounds, especially music... 



Some benefits and drawbacks... 


"Music activates similar neural systems of reward and 
emotion as those stimulated by food, sex and drugs/ R. 
Zatorre. Montreal Institute of Neurology. 
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Quantifying emotion 



Many formats for trying to quantify 
emotions, from sociological to 
neuropharmacological. 


Anger 

Rage 



Interest 

Excitement 


Enjoyment 

Joy 
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Active 

Aroused 



Passive 


Happy 

Pleased 


Simple 2 axis circumplex model based on arousal vs 
valence (Larson & Diener, 1992) 


None of them are entirely satisfactory 


"Emotion cube" model 
based on 
neurotransmitter 
release level correlation 
Lovheim, 2011 
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Auditory and attentional systems are deeply wired into 
emotionally responsive regions of the brain 
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Manipulation of sound enables us to trigger specific 
brain responses 
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Tiny changes in fine structure of sound has a 
huge psychological impact 






Pink noise (left) vs pseudorandomly 
amplitude modulated pink noise (right) 
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Rock your body: Sound affects physiology 



Auditory facilitation: 
entrainment of 
respiration and 
heartrate by sound. 


Sympathetic arousal - Pupillary Parasympathetic arousal (alternating 
dilation (3Fs) contraction) 




Time 


NS A: “Mr. Furious” 
Sympathetic 
stimulator 
(Wong, 2000) 


Start End 



Time 


NSA: “Afterglow...” 
Relaxer/Altered state 
inducer 
(Wong, 2000) 


Arousal? 
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So what can you DO to your victim player with 

proper sound? 



Unexpected physical 
effects: "Eyeball Twitch" 
modulation at resonance 
frequency of human 
eyeball (18.1-21 Hz) 
causes eye twitching and 
occasional visual illusions 



Emotional manipulation: 
"Ghost room" - near 
infrasonic distortion 
increases listener unease 
with repeated listening 



Make them MAD: 
"Mister Furious" 
Sympathetic 
nervous system 
activator 

(Fight/Flight Driver) 



Physical relaxation: 
"Oceanic" 

Auditory facilitation 
(breath/heart rate 
controller & 
relaxer) 



Make them sleep: 
"RealSIeep" - use sound to 
trigger vestibular systems 
that control and induce sleep 
and alertness (Sleep Genius) 



Make them feel like 
they're moving (or make 
them throw up). "Vertigo 
Tour" - trigger vestibular 
functions with LF sound. 
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Expectations of world model 

• Reinforce... or break and disappoint 

• Giving meaning to player verbs- power, 
agency, and mastery. 

• Audio as secret game design 
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Does audio make game visuals look better? 



Of course! Right?... 

Yes, but not quite like you think... 
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The Media Equation 


How People Treat Computers, 
Tele\ ision, and New Media 
Like Real People and Place?* 



Byron Reeves & Clifford \a*s 


The brain accepts AV input as reality, 
and treats it as such (!). 

Good audio -can- improve visual 
perception, but mostly... 

Poor audio quality can degrade 
visual perception 

Poor audio synchronization can 
degrade total perception 
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Actor Performance Consistency 

•VO as part of audio score... 
•Vocap 

•What if you can't vocap? 
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Talking about sound... 

- Fidelity (accuracy) versus inaccuracy 

- Naturalness versus awkwardness 

- Pleasantness versus annoyance 

- Signature audio identification 
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Signature Sound 

•Tactical 

•Branding 

•Expected 

•Unexpected 
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Audio and attention channels 

• Impressive sound and still be ineffective. 

• Give context, meaning, and a free channel. 

• Do not overwhelm. 
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Bulletstorm case study 

• The spatial dogpile... 
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The continuity illusion 




• One aspect of the wall of sound we try to fight is the Continuity illusion. Here's a 
visual example, where the red shapes look separate, but the same shapes in blue 
look continuous due to the green shapes between them. 

• You will hear short sine wave beeps, with white noise in the background. As the 
white noise gets louder, the beeps will start to sound like a continuous tone. 

• Too much continuity makes audio confusing and sound as a single audio event. 
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The rule of 2% and 'tuning' 



Encoded 


Encoded-Embodied 


Embodied 


SjQ 
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Strong response sources 



low sounds = scary 
Nails on chalkboard 




Some examples... 
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Friction Monster... 
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Speech Perception 

Leveraging speech prosody/melody 
But not in all cultures... 

Important considerations: 

• Spatialization 

• Masking between 200 and 5000 hz 

• Transmitted speech - intelligibility 


GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 



Human - Animal - Monster 
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Music In Games 


Some always turn off music... Why? 
. Foreshadowing too obvious 
. Distracting emotional beats 
. Cognitive dissonance 
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Emotional Response to Music 

Built-in: 

Evaluative conditioning 

Learned: 

Episodic memory 
Music expectancy 
Visual imagery 
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Nostalgia: Evaluative + Episodic 

Not imprinted, limited response: 

Imprinted, response: 


Play session style can inform how to approach 'earworm' goal 
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Audio in a VR Environment 

• Movie-Game-VR Contrast 

• HRTF issues and Azimuth Effects 

• Spatial relation and reflective environment 

• Single source observation (TV) vastly different 
than full VR presentation 

• Depth Congruency 

• Spatial panning... behind the listener? 

• Vestibular system issues 
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The Future 

• HDR + Meaning + Focus 

• From recording to generating. 

• Scientific audio ammo! UX too. 
. VR Audio 


Zak.belica@epicqames.com 

shorowitz@neuropop.com 


